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ABSTRACT 

This study sought to examine the nature and 
dimensions of prior knowledge among undergraduates in an economics 
course at the Open University of the Netherlands (OuN) . A total of 22 
law and 55 economics students enrolled in two economics courses were 
given a 154-item domain-specific knowledge test, which was then 
analyzed independently by three researchers. The researchers 
attempted to classify each of the 154 items on each of 10 dimensions 
(curriculum level, curriculum accent, node relation, behavioral, 
content, epistemological , number of propositions, information level, 
and representation level). The results of the analysis indicated that 
although different dimensions helped to differentiate between law and 
economics students, the different dimensions were not helpful in 
identifying more specific and. s igni f icant contrasts between both 
student groups. The study also found that the grouping variable 
"diploma type" was not able to differentiate between levels of 
mastery of the prior knowledge state. It is foreseen that in 
situations where there are significant differences between the prior 
knowledge state of specific subpopulation, the dimensions might be 
helpful to detect the strengths and weaknesses of the students 
involved. 
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1 Introduction 



There is no doubt that the prior knowledge state is playing a major role in the learning process of students. 
In our recent work (Valcke and Dochy, 1991; Dochy and Valcke, 1991b), the analysis of the quality and 
impact of the prior knowledge state has been a major focus. Several instruments have been developed to 
measure the prior knowledge state, especially within the domain of economics. In analyzing the prior 
knowledge state, we did especially focus on the structure of the prior knowledge state along a content 
dimension. 

In this report we report a study which supports the development of 'knowledge profiles' as an assessment 
tool in educational practice to direct future learning. 

In the theoretical part of this text, we discuss - in short - our distinct approach towards the analysis of the 
prior knowledge state 1 . This approach is based on an extensive analysis of the literature in relation to 
theories, models and practice-based strategies about the "structure of knowledge". This. base is exploited to 
define a set of "dimensions" that are helpful to construct "knowledge profiles". Four types of dimensions 
are illustrated : cognitive-psychological dimensions, educational-psychological dimensions, psychometrical 
dimensions and content-based dimensions. 

In the second part of this text, these dimensions are used to analyze the knowledge profiles of economics 
and law students. The results help to detect differences in the mastery of components of the prior 
knowledge state between both student populations and might be helpful to provide further evidence about the 
validity of the theoretical knowledge profile dimensions. 



2 Theoretical background 
2. 1 The structure of knowledge 

From an instructional-psychological point of view, the structure-of-knowledge problem should be 
investigated in order to find out more efficient ways for using instructional technology. Our search for 
means to deal with the prior knowledge state showed that one should take account different components of 
the prior knowledge state (Dochy and Valcke, 1991). The concept of "components" refers towards a 
structure in the knowledge base of the learner. 

Our earlier research was helpful to detect such components of the prior knowledge state along the content 
dimension. But it was also suggested that the differentiation of components of the prior knowledge state 
along other dimensions is needed to be helpful to diagnose educational practice (Dochy and Valcke, 1991; 
Dochy and Valcke, 1991b) 

The issue of the "structure of knowledge" has been debated from a variety of theoretical points of view : 
cognitive psychology, epistemclogy, philosophy, etc. At the more pragmatic level, the issue has also been 
of prime importance in applied sciences like instructional psychology, curriculum development theories and 
psychometry (Dochy, 1992). 

Disciplines like cognitive psychology, educational psychology, artificial intelligence, etc. - have - from their 
points of view - highlighted the "structure of knowledge" resulting in a puzzling variety of approaches, 
models and (Ausubel, 1968, de Groot, 1946, Mayer, 1979, Reigeluth and Stein, 1983). 



A more elaborated version of the theoretical ban of the knowledge profile dimensions can be found in : Dochy & Valcke (1 99 1 a). OTIC. 
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It should be noted that our primary focus in using these theories originates from an information processing 
view on learning (Sternberg, 1985a & 1985b). The main reason for this is that we stress a dynamic 
approach towards the structure of knowledge, which is in particular advocated in this view. If we 
summarize the variety of approaches, four main types of dimensions to structure knowledge can be 
conceptualized : 



Content related dimensions 
Cognitive-Psychological dimensions 
Educational-Psychological dimensions 
Item Characteristics dimensions 



2.2 Knowledge profiles 

As such, the concept of 'knowledge profiles' is not found in literature. Only 'student profiles' (Wolf, et. 
al., 1991) and cognitive profile' (Letteri et. al., 1982) have some simi'arity in meaning. This is certainly 
the case for the studies by Letteri et. al. (1980, 1982). The concept 'profile' is derived from the practice, 
common in educational research, of plotting as a graph or profile the scores of a person as raw scores or as 
standardized scores (Keeves, 1988). In analyzing research findings, comparisons are made between persons 
or groups in terms of a set of measurements on specific related aspects. For each person or group a profile 
is obtained on a set of parameters. The comparison between profiles of persons is known by the generic 
term 'profile analysis'. 

Figure 1 shows the relationship between some key concepts. A "dimension* is used to construct a 
knowledge profile. Each dimension, consisting of several parameters, represents an approach towards the 
structure of knowledge. 




Miumi 



Figure 1: Example of a profile 

From an instructional psychological point of view, knowledge profiles can give practical indications of 
student achievement and learning in order to direct the learning process. In a recent overview of student 
assessment, Wolf et. al. (1991) advocate this approach. According to these authors, there is a need for a 
new brand of educational psychometrics capable of answering the much changed questions of educational 
achievement. These changes are the new premises, the multiple paths towards the prior knowledge state, 
more developmental oriented assessments and the ascertainment that students enter school with widely 
varying backgrounds. In our terms, we take account of these changes by trying to identify multiple 
components of the prior knowledge state, by implementing prior knowledge state tests and by intending to 
use these tests as progress tests administered several times a year. In this context it is necessary to come to 
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an agreement on the relevant parameters to describe student performance and it is critical to develop ways 
of looking at -'student profiles': "unless we- develop these kinds of differentiated portraits of student 
performance within a domain, it is difficult to envision student assessment ever informing, rather than 
merely measuring, the educational process'' (Wolf, et. al., 1991). 



2.3 Overview of knowledge profile dimensions 

Only those dimensions/parameters are reviewed that have been retained after their discussion and analysis in 
our earlier publication "Validation of Knowledge Profile Dimensions : Looking for empirical Evidence" 
(Dochy and Valcke, 1991a). 

If dimensions are based on a model or theory, only short details will be reported. We will shortly report on 
I je models or theories on which dimensions are based. The first dimensions are classified according to 
common models of economics. Other dimensions are based on theories on knowledge representation and on 
knowledge structure, learning theories, text representation models and psychometric theory. 

2.3.1 Content related dimensions 

Economics subdomains dimension 

"Content" is one of the most exercised dimensions to categorize domain knowledge. The classification based 
on the parameter 'subdomains' refers to the subdivision of the economics-domain into "subject matter 
blocks" that are common within the science of economics. Our dimension structure, as implemented in the 
curriculum structure of the University of Maastricht, contains nine parameters : 



1 . Reporting 

2. Financing 

3. Organization 

4. Marketing 

5. Macroeconomics 

6. Micro-economics 

7. Public finances 

8. International economic affair* 

9. Behavioural and social science* 



Curriculum level dimension 

Some parts of the content of a science are supposed to be mastered by the students at certain moments 
during their study. These moments are called the curriculum levels (first and second year). These levels 
are subsequent, but too broad to be supposed hierarchical. 



1 . First year level 

2. Second year level 



Curriculum accent dimension 

Within economics it is common to differentiate between two main streams, representing a different accent, 
i.e. general economics and business administration on the one hand and quantitative economics on the other 
hand. 



1 . General economics and business 

2. Quantitative economics 
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2.3.2 Cognitive psychological dimensions 
Node relation dimension 

Knowledge representation, as used in schema theories (Dochy and Bouwens, 1990), takes certain 
propositions or nodes as a starting point. A proposition is the smallest unit that can be qualified as true or 
false in a statement. According to most schema theories there are five kinds of nodes : Physical State (PS, a 
statement that refers to an ongoing state in the physical or social world), Physical Event (PE, a statement 
that refers to a state change in the physical or social world), Internal State (IS, a statement that refers to an 
ongoing state of knowledge, attitude, or belief in a character), Internal Event (IE, refers to a state change in 
knowledge, attitude or belief in a character), Goal (G, a statement that refers to an achieved or unachieved 
state that a person wants) and Style (S, a statement that refers to details about the style or manner in which 
an action or event occurred. 



1. G - G REASON 

2. PS-G INITIATE 
IS - G 

PE-G 
IE - G 



3. PS - PE CONSEQUENCE 
IS-PE 
PE-PE 
IE-PE 
G - PE 
PS-PS 
IS - PS 
PE-PS 
IE-PS 
G -PS 



4. PE-S/G MANNER 
EE - S/G 
GE - S/G 



5. PS - PS PROPERTY 



The "Node Relation" dimension is based on characteristics of the interrelations between propositions, called 
node relation or arc parameters: Reason (R, a Goal node is a reason for another Goal node), Initiate (I, a 
State or Event initiates another Goal node), Consequence (C, a State, Event or Goal node that has the 
consequence of another State or Event node), Manner (M, an Event or Goal node occurs with some style), 
Property (P, a person, object or entity has some property that is a State node) (see also Dochy and 
Bouwens, 1990). These arc parameters are not hierarchical in nature. 

2.3.3 Educational-psychological dimensions 

The theoretical base of these two 1 dimensions - i.e. behavioural and content dimension - is found in 
Component Display Theory (CDT, Merrill, 1983), Taxonomic theories (De Block, 1986 and Bloom, 1976) 
and Gagn6's theoretical classification (198S). 

Behavioural dimension 

The known distinction between declarative and procedural knowledge is further operationalised at this stage 
into the parameters 'to know, to understand, and to apply'. These parameters are also perceived as 
equivalent to the concepts 'recognition, reproduction and production'. Items can be classified as measuring 
the appreciation, the recognition and the reproduction of information (declarative) or measuring production 
or applications (interpretative, convergent, divergent or evaluative production)(procedural) (Keeves, 1988). 
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The three parameters do also correspond with taxonomic levels proposed by several educationalists as 
Bloom, Guilford, De Corte and De Block (cf. Keeves, 1988). Most researchers agree that these parameters 
are of a hierarchical nature. 



1. Know 


1. Declarative 


2. Understand 




3. Apply 


2. Procedural 



Content dimension 



Along the content dimension we differentiate between five parameters : facts, concepts, relations, structures 
and methods. This is in accordance with e.g. the work of Guilford when he refers to product parameters 
(Keeves, 1988). These parameters are also widely accepted as being hierarchical (Keeves, 1988). 

1 . Facts 

2. Concepts 

3. Relations 

4. Structures 

5. Methods 



Epistemological dimension 



Based on the levels of knowledge representation of Brachman and Schmolze (198S), five parameters can be 
differentiated along a typical dimension. These parameters can also be considered as the most appropriate 
combinations of behaviour- and content dimension parameters, as clarified between brackets : knowledge 
identification (identifying facts and concepts), knowledge conceptualisation (insight in concepts), 
epistemological analysis (to know and understand, relations and structures), logical analysis (to know and. 
understand methods), implementational analysis (application of methods). These dimensions are considered 
as hierarchical since they are a combination of the hierarchical behavioural and content dimension. 



1 . Knowledge identification 

2. Knowledge conceptualisation 

3. Epistemological analysis 

4. Logical sna lysis 

5. Implementational analysis 



2.3.4 Item characteristics dimensions 



Number of propositions dimension 

A proposition is the smallest unit that can stand as a separate assertion which can be judged as true or false. 
In schema theories (Dochy and Bouwens, 1990), propositions or nodes have a core function in the structure 
of schemata. It is assumed that the amount of propositions determines the degree of structure needed to 
answer the item correctly. Three parameters have been identified in relation to this dimension : 



1. < 5 propositions 

2. > 4 < 10 propositions 

3. > 9 propositions 



Information level dimension 



The "stem" of an item is the general information which is given and which must not be evaluated. This 
correct information precedes the questions for which this information should be taken into account. A stem 
can be connected to one or more subsequent questions. 
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Since the spatial and logical distance between the general information part of an item and the question part 
is larger in items with a stem than for items without a stem, the difficulty level of the former is expected to 
be higher. 

1 . Items with a stem 

2. Items without a stem 

Representation level dimension 

Following the classification used in the kesearch of Boekaerts (1979), i.e. visual, verbal and symbolic 
representation, we distinguish four parameters along this dimension. These parameters are also closely 
related to the four content levels of Guilford's structure of the intellect: figural, symbolic, semantic (the 
verbal factor) and behavioural (nonverbal information) and the Twyman (198S) categories : verbal, pictorial 
and schematic. 

Test-items are always based on textual information representation, but can be enhanced, enriched or 
documented with information of an other representation category : 



1 . Textual-graphical 

2. Textual 

3. Schematic 

4. Textual-symbolic 



2.4 The prior knowledge state of economics students (ES) and law students (LS) 

The research population, as explained in the next paragraph, involved both economics an law 
students. Earlier research (Dochy and Valcke 1991; Dochy and Valcke, 1991b; Wagemans, Valcke and 
Dochy, 1991) helped to put forward the following conclusions : 

single variables, such as diploma type (e.g. economics, law), are not good indicators of the 

prior knowledge state; 

ES and LS are different in relation to specific components of the prior knowledge state; 
there is a trend that ES perform better that LS, but this difference is not statistically 
significant. 

In the next part of this text, the 'knowledge profiles', discussed earlier, will be used as an assessment tool 
to compare in greater detail differences among the two student populations. 



3 Research design 

3.1 Hypotheses 

Taking into account the theoretical base of the present study, the following main hypothesis can be stated : 

"Economics students (ES) and Law students (LS) are not different in terms of the variables alontf a 
variety of knowledge profile dimensions. " 

Since up to 10 profile dimensions will be used, the main hypothesis can be split up into a set of 10 
subhypo theses : 

- ES and LS are not different in terms of the economics subdomain knowledge profile. 

- ES and LS are not different in terms of the curriculum knowledge profile. 

- ES and LS are not different in terms of the curriculum accent knowledge profile. 

- ES and LS are not different in terms of the node relation knowledge profile. 

- ES and LS are not different in terms of the behavioural knowledge profile. 

- ES and LS are not different in terms of the content knowledge profile. 
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- ES and LS are not different in terms of the epistemological knowledge profile. 

- ES and LS are not different in terms of the representation level knowledge profile. 

- ES and LS are not different in terms of the number of propositions knowledge profile. 

- ES and LS are not different in terms of the infoi ^nation level knowledge profile. 

3.2 Research instruments 

A domain specific knowledge state test (PKS) was administered to the research population. This test 
consists of 154 items. The test covers the whole domain of ecc iomics to be studied at university level in 
relation to the courses "Economics & Money" and "Balance sheet, Profit and Loss Account and 
Administrative Procedures". This test consists of multiple-choice questions which can be answered with 
true/false or ?. The ? -alternative is taken as a third alternative in order to prevent guessing. 
Characteristics of the test suggest that the determination of certain psychometric qualities might be a 
problem. There is no problem in relation to validity since the test clearly represents - to a very large extent 
- the domain and since the test has been developed by a team of domain experts. 

Table 1 : 



a-coefficients for the course subtopics and curriculum 
accent dimensions and mean a-coefficients 



PARAMETERS 


a 






Reporting 


.5739 


18 




Financing 


.6449 


18 




Organization 


.6922 


18 




Marketing 


.6292 


18 




Macro-economics 


.7069 


25 


.631 


Micro-economics 


.7420 


25 




Public finances 


.5101 


11 




Intern, economic affairs 


.5543 


11 




Behavioural & social sciences 


.6287 


10 




General economics & B.A. 


.9270 


139 




Quantitative economics 


.4467 


15 


.686 



On the other hand, determining the reliability of the test induces some specific problems. If we calculate 
the alpha-coefficient, the test can be considered as reliable: a = .9302. But this high reliability level is 
marred by the fact that the test is very long (154 items); thus resulting rather easily in a high o-coefficient. 
More important, calculation of the a-coefficient supposes the test to be homogeneous. Mostly tests are tests 
are homogeneous at the content level. The delineation of the knowledge profile dimensions above, indicates 
that this basic assumption to calculate the a-coefficient has been violated. A solution to this problem might 
be to check the reliability of subparts of the test, making use of the knowledge profile dimensions. 
Calculation of a was repeated for two of these dimensions (course subtopics and curriculum accent), in 
order to be able to present a mean reliability score. After reorganising the test into more homogeneous 
subparts, a-coefficient and a mean a-coefficient was calculated. The results of this procedure are 
summarized in table 1. To be able to judge the figures in a better perspective, the number of items each 
subgroup of items consists of is also given. 

Mean a seems to be > .63. This reliability score is - taking into account the restricted number of items in 
certain subparts of the test - acceptable for our research purposes. 
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3.3 Research population and procedure 

3.3.1 Research population and sample size 

The research population consisted initially of 91 students. This number of students resulted from a 200 
students containing, much bigger random sample, who were invited to participate in the project after 
subscribing for the course "Economics & Money" or "Balance sheet, Profit and Loss Account and 
Administrative Procedures" at the Dutch Open University (OU). The two courses are part of the 
compulsory program of different diplopia lines, such as "Dutch Law" and "Economics". When 
administering the test to the students additional information was gathered in order to define of each student 
his diploma intentions. This helped to divide the research population sample into three groups : 55 
economics students (ES) and 22 law students (LS) and a third group consisting of 14 students following 
other or undefined diploma lines. The latter group of students was withdrawn from the research sample. 
Final sample size was therefore 77 students. 

3.3.2 Research procedure 

The domain specific knowledge state test was administered to the sample of Ou-studects. The raw scores 
for the test items were receded in order to gather a maximum of information in relation to mastery or non- 
mastery of the domain specific knowledge. After receding 2 , a general economics-score for the entire test 
'./as calculated. 

In a next step, all items were classified along the dimensions discussed in part 2 of this text. The 154 items 
were analyzed - separately - by three researchers. In reviewing the items, the researchers attempted to 
classify each item on each one of the 10 dimensions. An inter-rater reliability was obtained > .8T 3 . If 
there was discussion in relation to the categorization of a specific item along a dimension, discussion 
resulted in a consensus on the final evaluation of the item. 

Grouping the items along the knowledge profile dimensions helped to calculate specific subscores. To 
compare the mean total subscores, the individual subscores have been calculated as %-scores. 

4 Research results and discussion 
4.1 General results 

Table 2 on the next page gives an overview of the mean scores and subscores for the entire test and the 
different regroupings of items along the 10 dimensions 4 , named in the first column 

Next the names of the different parameters along the dimensions are recited, with - in the third column - the 
number of items that have been identified as exponents of this parameter. The mean % score of the total 
research sample for each specific parameter is reported in the fourth column. 

The mean % scores for the different parameters in relation to each dimension show sometimes striking 
differences. The subtopic dimension presents for instance mean % scores varying from 18.3 % to 44.37 %. 
This suggests - a first level - that some dimensions/parameters can help to indicate mastery or non-mastery 
of components of the prior knowledge state. 

Next to the differences in mean scores, especially the large a-values draw our attention. These large values 
are the result of the fact that the test measures " the prior knowledge state" of students with a wide variety 
of prior experiences in relation to the topics assessed by the test. 

Another striking fact is the large difference in the number of items that help to calculate the parameter- 
subscores. Some N^-values are even problematic. The "Reason" parameter along the "Node Relation' 



1 The normal scoring procedure for this test implies thtt students obuin + 1 when their answer is correct; obuin 0 when they tnswer with 
? and obuin -1 when their inswer is wrong. In the recoding process, scoring for wrong tnswers wts changed into a zero-score. 

1 • -p £.05 •• -p £.01 

4 The concept 'significant* in this report refers to "statistically significant* results 
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dimension is e.g. represented by only two items, making this parameter less useful and weakening the 
validity of this dimension. This has to be taken into account when interpreting the results of the analysis. 



Table 2 : 
Overview of general results 



DIMENSION 


MMionu 


\_ 




m 


IC*IMMlC»M)M0IMlM 


Keen** 


ts 


38.3! 


18.28 




Fnmdnj. 


IS 


33.26 


18.96 




Orgeaoatian 


IS 


44.37 


21.15 




MwkMmf 


IS 


42.78 


21 .12 






25 


2135 


18.52 




111,1., ,111, t 


25 


26.13 


2Z37 






u 


MM 


21 .31 




faikras* #©eo0tnicB iifun 


U 


11.30 


20.09 




Bcfasviounl A tocicl tcieeoc* 


10 


19.61 


20 J9 


Curriculum lm( 


Fir* >w level 


102 


30.49 


15.08 




Second ynr tovcl 


52 


31.22 


17.74 


CturtculiM* >««al 


Gcuoitl (POBPOottDt A 9*A. 


15 


27.01 


16.22 




Quantitative •oomnks 


139 


31.14 


16.14 


Hot* relation araiMUn 


ftmtm 


2 


22.73 


28.73 




tttkle 


S 


3Z14 


21.11 




Coemcjubood 


58 


30.47 


15.95 




Uw 


5 


42.60 


23.47 




Property 


SI 


30.25 


16.33 


MHvtoural lewl 


Know 


24 


29.35 


17J6 




UnfentenJ 


99 


31 J» 


15.91 




Apply 


31 


30.00 


16.83 


Cwttoit level 


F«ot 


6 


33 JO 


22J2 




Conoepu 


21 


3S.47 


19.08 




Rektiam 


32 


29 M 


16.14 




Suuotueee 


56 


26.97 


16.59 




Method* 


39 


32.40 


17.19 


KpoitoeiMtoflcfJ 


KaaMledao UcaUkatkxi 


15 


34.63 


1938 




Knowledge ooacoptueliaetian 


10 


43.25 


20.93 




Eplmmelotlotl maiyilt 


S4 


27.94 


16.04 




Lacked emlyele 


15 


35.84 


19.68 




lavfaaHOUtkml emlyele 


30 


29.91 


17.00 


Number «f prepeelHew N 


< 5 


S2 


29.08 


15.66 




> 5 < 10 


44 


14.44 


07.49 




> 9 


21 


09.05 


05.13 


Infermfttien level 


hem with Hem 


105 


33.39 


17.18 




hmm without Mem 


49 


42.41 


23.16 


K eerie en tetton level 


TettueJ-gnphkal 


99 


32.00 


16.27 




Tonal 


9 


29.00 


22.74 




Sohemetlc 


14 


3Z6S 


18.41 




Symbolic 


32 


27.84 


17.83 


ECONOMICS TKSTSCORK 




154 


47.34 


24.14 



BEST COPY AVAILABLE 




ummmsiMiM 



Knowledge profile! of economic! and Uw itudenu 



4.2 Profiles ot ES & LS : a first analysis 



Mean % agora* 




Economics 



Figure 2 : Mean economics scores of ES and LS 

Figure 2 depicts the differences in the mean % scores of ES and LS for the overall economics score. This 
difference is - although nearly 10% - not statistically significant (F=2.124, pp=.149) as expected. But as 
suggested in earlier research reports (Dochy and Valcke 1991, p. 11), a more thorough analysis of the overall 
economics-score for the PKS-test can be helpful to reveal specific (and significant) differences. 

4.2.1 Economics subdomains dimension 
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Figure 3 : Economics Subdomains knowledge profile 

The data in figure 3 reveal clear differences in the mean scores of economics and law students for the different 
economics-subdomains. The mean % scores of ES are higher for most subscores, with the exception of "macro- 
economics", "public finances" and "behavioural & social sciences". If we neglect the interrelations between 
the different economics-subdomains, we can test the significance of the differences between the mean-scores 
by a univariate F-test 5 . This analysis of variance reveals that the differences in mean % scores of ES and LS 
are not significant. The most important pp-value is obtained in relation to "Marketing" (F=4.554, pp=.036). 



3 In part 5 .3 of this text, we will take the iniercorrelation between the subdomains into account when executing a multivariate analysis of 
vsrisnce. A summary of univariate analysis of variance results is given in psrt 4.2. 1 1 . 
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4.2.2 Curriculum level dimension 
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Figure 4 : Curriculum knowledge profile 

As expected, the mean score for the level-2 items arc lower for both sub-populations. It is normal that the prior 
knowledge state of these advanced level questions is restricted. 

Although economics-students always obtain higher mean % scores for both course levels, these differences are 
not statistically significant. 

4.2.3 Curriculum accent dimension 
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Figure 5 : Curriculum accent knowledge profile 

The curriculum accent profile shows that "General economics & Business Administration" mastery is higher 
than "Quantitative economics" mastery. Also interesting is the fact that the difference between ES and LS is 
greater in relation to "General economics". Nevertheless this difference remains non-significant. 
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4.2.4 Node relation dimension 
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Figure 6 : Node relation knowledge profile 

There seems to be hardly a difference in the mastery of the "Reason" node relation category. But as 
commented in the former part of this text, this parameter is of little relevance due to the restricted number of 
items on which the scoring is based. The biggest difference between the mean % -scores of ES and LS is 
observed is association with the "Initiate" and "Manner" node relation category. Only the latter difference is 
statistically significant, but at the 5%-level (F=3.755, p,r=.05). 



4.2.5 Behavioural level dimension 
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Figure 7 : Behavioural knowledge profile 

At the theoretical level, it is expected that differences, found at a lower behavioural level, have an impact on 
differences in relation to higher behavioural leve's. Figure 7 gives support to this hypothesis since initial 
differences on the "Know" -level widen to bigger differences at the "Apply "-level. But the differences between 
ES and LS remain non-significant. 
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Figure 8 : Content knowledge profile 

The "Content level" knowledge profile is also based on a hierarchical categorisation of domain knowledge. 
Figure 8 presents a peculiar prior knowledge state profile of ES and LS. "Factual" knowledge seems to be 
higher for LS. But from this level on, the mastery of ES is always superior. This can be explained : ES have 
already integrated the factual knowledge in more complex structures or schemes (e.g. concepts, relations, 
structures). Therefore, the immediate availability of the isolated knowledge elements is less elaborated than for 
law students. But the integrated factual knowledge is more readily available and more functional. In the 
knowledge profile, the differences between ES and LS are the biggest at the "Concept "level. This difference 
is only slightly significant (F=4.729, pF~.03). 



4.2.7 Epistemological level dimension 




Figure 9 : Epistemological knowledge profile 

Although the knowledge profile of ES is consistently higher than the profile of LS, the differences in mean % 
scores are non-significant. 
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4.2.8 Representation level dimension 
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Figure 10 : Representation level knowledge profile 

The mastery of representation levels suggests that LS are more able to solve questions based on graphical 
information. ES seem to master questions, based on symbolic information to a higher extent. Although 
interesting, none of these differences in mean % -scores are statistically significant. 

4.2.9 Amount of propositions dimension 
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Figure 11 : Number of propositions knowledge profile 

Items with a high number of propositions are more complex than items with a low number of propositions. It 
is expected that ES perform better at all proposition-levels. The profiles in figure 1 1 give support to this 
hypothesis, but statistical analysis does not reveal significant differences in mean % scores of ES and LS. 
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4.2. 10 Information level dimension 
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Figure 12 : Information level profile 



Mastery of items without a stem seems to be slightly higher than mastery of items with a stem. This can be 
related to the findings in relation to the number of propositions.Items without a stem are based on a smaller 
amount of propositions and are therefore seemingly more easy to comprehend and to solve. Although ES 
perform better in relation to both types of questions, these differences in mean % -scores are not statistically 
significant. 

4.2.11 Intermediate conclusions 

Table 3 on the next page gives an overview of the results of the univariate analysis of variance when comparing 
the mean % scores of ES and LS*. Although the different dimensions help to differentiate between ES and LS, 
the different dimensions are not helpful to identify more specific and significant contrasts between both student- 
groups. This affirms our earlier research findings showing that "student type" might not be a relevant 
"indicator" of the prior knowledge state (Dochy and Valcke, 1991, Valcke and Dochy, 1991). The initial non- 
significant difference between both sub-populations (overall economy-score) is confirmed, but again the general 
trend that ES perform better than LS can be repeated. 

4.3 Profile analysis 

An univariate analysis of variance (as used in part 4.2 of this text) does not take into account the 
intercorrelations between the different parameters along the profile dimensions. These intercorrrelations are 
important (although not making the specific variables redundant) and can be explained at the theoretical level 
as clarified elsewhere (cf. Dochy and Valcke, 1991a). A multivariate analysis of variance is needed to refine 
our analysis and to look for more conclusive information about the differences in the prior knowledge state 
between ES and LS. A multivariate analysis can take these intercorrelations into account. Profile analysis is 
an extension of multi-variate analysis and is especially appropriate and helpful to evaluate the parameter 
structure in relation to each profile dimension when comparing subpopulations. Several tests are available in 
profile analysis. Of principal interest - for our purposes - is the "parallelism 11 test which help to answer the 
question whether the profiles of two subpopulations are parallel or not. At the theoretical level, also the 
"flatness" test might be relevant, since this test controls the similarity of responses for the different parameters 
along a dimension, independent of groups or subgroups. An answer to this question helps to support the validity 
of the different dimensions since the results indicate whether or not the dimensions/parameters are helpful to 
specify differences in the mastery of different components of the prior knowledge state. A profile analysis will 
be performed on the complex of parameters in relation to each dimension. 



* Since variances in both subpopulations are homogeneous, univariate analysis of variance can be applied. 
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The grouping variable is "diploma type" (ES or LS). SPSS-PC + MANOVA was used for our profile analysis. 

Table 3 : 
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4.3.1 Control of underlying assumptions 



Profile analysis implies that specific assumptions about the quality of the research data are met (no missing data, 
comparable sample sizes, (multivariate) normal distributions, no outliers, homogeneity of variance-covariance, 
multicollinearity). 

Data screening revealed no missing data. 

Although sample sizes are different for both subpopulations (N ES =55 ; 1^=22) no special difficulties 
are expected since only one independent variable is used. 

The evaluation of the homogeneity of variance-covariance matrices is based on the Cochrans C and the 
Bartlett-Box F test. 

To evaluate assumptions about multivariate normality, boxplots of the mean submeasures for each 
dimension have been screened. 

Multicollinearity is tested with the Bartlett test of sphericity. 



ERIC 



22 



Knowledge profile* of economics and Uw students 



In evaluating multivariate normality of the distribution of the mean % -scores, it is to be mentioned that the 
<r-values are very high, indicating a wide dispersion of the scores. This is to be expected, since the test 
measures the "prior knowledge state". Figure 13 presents e.g. a box-plot of the scores of ES and LS for the 
subtopic "marketing" on the "Economics subdomain" dimension. The wide dispersion of the scores is obvious. 
The * identifies the median and the box contains the middle 50% of the values. 

The lines emanating from the box extend to the smsllest and largest observations in the subgroups that are less 
than one interquartile range from the end of the box. Points outside this range are marked with O (outliers) 
or E (Extremes) if more than l.S interquartile distances away from the box. 
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Figure 13 : Box-Plots 



Analysis of the box-plots for each variable in relation to each profile dimension reveals that there are outliers 
and extremes, but that their number remains restricted. 

Table 4 on the next page summarizes the data in relation to the evaluation of the homogeneity of variance- 
covariance matrices and the multicollinearity test. 

Only in relation to one dimension, the assumption in relation to the homogeneity of the variance-covariance 
is violated ("curriculum accent" profile dimension). 

The Bartlett test of sphericity is significant in all cases, which means that the variables are highly 
intercorrelated. Although the p-values are very small, the SPSS-MANOVA-PC + procedure protects against 
instability caused by multicollinearity by excluding variables from the analysis with too low tolerance levels 7 . 
The fact, the MANOVA-procedure was never halted during execution indicates that multicollinearity did not 
cause problems. 

In general we can conclude that assumptions are met in order to execute a profile analysis on the research data 
available. Only, in interpreting the analysis results in relation to the "curriculum accent" dimension we will have 
to take care. 



1 Tolersnce level m 1 - SMC (squared multiple correlation of each variable). 
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TabU 4 : 

Analysis data in relation to multicollinearity 
and homogeneity of variance-covariance matrices 
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4.3.2 Profile analysis results : parallelism test 

Table 5 presents an overview of the analysis results in relation to the parallelism test. This helps us to answer 
the question whether the two different student groups have parallel o:: non-parallel profiles. This s commonly 
known as the test of parallelism and is the primary question addressed by profile analysis*. In relation to each 
profile dimension, Wilk's Lambda (X) was calculated and p-levels determined. In the results table, Wilk's X 
is not reported in relation to three dimensions (marked with •). This is because these dimensions only contain 
two variables; in these cases a test of significance for Hoteling's T 2 using the unique sums of squares was 
calculated, checking the interaction of the independent variable (diploma type) and the two dependent variables 
on the specific dimensions. For these cases the F-value and pp-value are reported. 

Table 5 : 
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If the multivariate analysis of variance indicates significant differences, the latter analysis can be extended by 
calculating structure coefficients 9 to determine the discriminatory power of the separate values on each profile 
dimension. This extra information will be embedded in the discussion of the table content. 

The data in table 5 are not helpful to detect more specific significant differences between ES and LS. The 
profiles of ES and LS are parallel. The intermediate conclusion of non-significant differences between ES and 
LS, based on analysis of the overall economics-score, cannot be revisited by a more refined analysis, based on 
the 10 dimensions.. The only, slightly significant F-value is obtained in relation to the content-level dimension 
(p= .025). When looking at the univariate analysis of variance and the mean scores of both subpopulations in 
fig. 8, it is perceivable that the potential differences in the content profiles are especially caused by the 
differences in the mean % scores for test-items measuring the mastery of "concepts". But a significance level 
of p = .025 is - in our opinion - insufficient to continue our analysis. 

Since the results indicate no significant differences at this level, a more elaborated profile analysis or the 

calculation of structure coefficients is not relevant. 

As a consequence, the hypotheses stated in part 3. 1 cannot be confirmed. 

4.3.3 Profile analysis results : flatness test 

Is the mastery of the prior knowledge state as defined by the parameters along a dimension different, 
independent of the groups (a within-subjects main effect) ? In other words, do students master the prior 
knowledge state in a similar way as defined by the different parameters along a dimension ? This question is 
now especially relevant since the profiles are parallel, as has been found above. 

If the flatness test is non-significant, then the profiles are not helpful to clarify or detect differences in the 
mastery of different components of the prior knowledge state. 

The results of the flatness test are therefore of relevance in relation to the validity of the knowledge profile 



' When tiling profile analysis as a substitute for univariate repeated mea*urei ANOVA, the parallelism teat ii the tett of inunction. 

* Since the aubvtluei on each profile dimension are highly ir'ercorrelated, we cannot viae raw or standardized discriminant function 
coefficients. The highly correlated variables 'shsre' the discriminants weights. It is safer to base our interpretation on the structure 
coefficients which are less likely to be influenced by these intertorrclationa. 
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dimensions. 

The results of the flatness test are found in table 6. For each dimension Wilks X has teen calculated, vitli the 
exception of the three dimensions where only two parameters are available along the dimension; there the F- 
value is reported (marked with *). 

Table 67 

Results of the flatness test in profile analysis 
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With the exception of the "curriculum accent" and "behavioural level" dimension, all dimensions/parameters 
help to differentiate in the mastery of components of the prior knowledge state. The non-significant results for 
the "curriculum accent" dimension can be explained by the restricted number of items, measuring the mastery 
of general economics and B.A. The differences in the mastery of the prior knowledge state along the 
"behavioural level" dimension is indeed very small and as expected non-significant (cf. table 2). 



5 Conclusions 

In this text, we attempted to analyze the prior knowledge state of two specific Open university subpopulations, 
studying an economics course. Earlier research helped to confirm that prior knowledge state differences do 
exist between economics and law students. In the present study, the overall economics score was not 
significantly different between both student populations. But in this research, special attention was paid to a 
further elaboration of this general economics-score by grouping items along a variety of knowledge profile 
dimensions. The theoretical base of such dimensions was briefly outlined. In analyzing the research data, 
profile analysis was used as a specific extension of multivariate analysis of variance (parallelism test). The 
results of the analysis indicate that the grouping variable "diploma type" is not able to differentiate between 
levels of mastery of the prior knowledge steie. Although this profile analysis - focusing on the parallelism test - 
could not help to reveal specific significant differences between the two subpopulations, the present study is 
of high importance since we succeeded in defining and operationalising a new more promising approach towards 
the analysis of the prior knowledge state. It is foreseen that in situations where there are significant differences 
between the prior knowledge state of specific subpopulations, the dimensions might be helpful to detect and 
dissect the strengths and weaknesses of the students involved. This might be a promising starting point for 
differentiated diagnostic and guidance approaches. 

The relevance of the knowledge profile dimensions was confirmed by the results of the flatness test during the 
profile analysis. Most dimensions help to differentiate in the mastery of specific components of the prior 
knowledge state. 
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